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Abstract: In this paper we present an app framework for augmenting large-scale walkable maps and orthoimages in 
museums or public spaces using standard smartphones and tablets. We first introduce a novel approach for using huge 
orthoimage mosaic floor prints covering several hundred square meters as natural Augmented Reality (AR) markers. 
We then present a new app architecture and subsequent tests in the Swissarena of the Swiss National Transport Mu- 
seum in Lucerne demonstrating the capabilities of accurately tracking and augmenting different map topics, including 
dynamic 3d data such as live air traffic. 

The resulting prototype was tested with everyday visitors of the museum to get feedback on the usability of the AR 
app and to identify pitfalls when using AR in the context of a potentially crowded museum. The prototype is to be 
rolled out to the public after successful testing and optimization of the app. We were able to show that AR apps on 
standard smartphone devices can dramatically enhance the interactive use of large-scale maps for different purposes 
such as education or serious gaming in a museum context. 
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1. Introduction 

Augmented Reality (AR) has been on the verge of becoming a major factor in a rapidly developing market for 
mobile communication. In marketing, AR has already become a widely used asset, for example to enhance printed 
material with interactive features. AR is serving as an enhancement to the perception of a user to the real-world 
providing virtual content without immersing the user completely into a virtual environment (Azuma 1997). The prin- 
ciple of any virtual augmentation fits within the reality- virtuality continuum (Milgram et al. 1995) and changed from 
purely conceptual experiments to practical solution, in that regard the increased usage of smart mobile devices pro- 
vided a game changer. It permits to move away from rather experimental approaches to virtual augmentation that 
make extensive use of complex interacting pieces of hardware that would rather not suit the everyday life behavior of 
potential interest groups as shown in Fig. 1. 
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Fig. 1. Concept of a head mounted AR system (Caudell & Mizell Fig. 2. Using smartphones for augmenting virtual contents on the 
1 992 ) real world (Christen et al. 20 1 6) 


Considering the speed of development in performance on mobile hardware and the growing need of users for con- 
venient access to information of interest, the rise of concepts employing AR technology is evident. Consequentially, 
a completely integrated and ready-to-use device with exceptional capabilities is now at our disposal to create new 
ideas for virtual augmentation and interaction solutions like augmenting archeological panorama images onto the real 
world as shown in Fig. 2. (Christen et al. 2016) 

In this paper we lay focus on specifically using geo spatial data in the context of a museum, present a practical 
approach to an AR app for mobile devices and discuss integrated contents. 


2. Related work 


A museum represents a place of audio visual experiences for its visitors. Using technology to enhance these expe- 
riences had a growing relevance over the past 20 years. One example to be mentioned here is from Wojciechowski et 
al. (2004). The authors introduce a vision-based augmented reality installment to show 3d reconstructions of destroyed 
or partially destroyed archeological objects. While a very practical approach was presented, the hardware consisting 
of webcams and displays seemed still a bit too impractical for daily use. 

The idea of using augmented reality in the context of a museum as an enhancing tool for visitors and as a guiding 
system was also demonstrated by Miyashita et al. (2008) to provide a system accessible by the visitors. The authors 
used a locally installed system consisting of a webcam attached to a cable-bound display to examine details of the 
artworks as shown in Fig. 3. The available hardware posed limitations to the mobility and flexibility of the system as 
well as limited concurrent viewers of the same artwork at the same time. 



Fig. 3. AR presentation system in a museum (Miyashita et al. 
2008) 


Fig. 4. Augmenting 3d terrain models over ordinary paper maps 
(Loesch et al. 2015) 
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Using geo spatial data to create AR is presented in Fig. 4. An ordinary paper map was used to augment a 3d digital 
terrain model onto using visual AR tracking. Results from one of our early projects called GeoAR (Geospatial aug- 
mented reality) (Loesch et al. 2015). 


3 The Swissarena - A country on one map 

The Swiss National Transport Museum in Lucerne, Switzerland records roughly 500 ’000 visitors each year and is 
among the most visited museums in the country. The Swissarena exhibition hall houses an approximately 200 square 
meters sized orthoimage of Switzerland at a scale of 1 : 20 ’000 (see Fig. 5). The ground sampling distance of the image 
is 30 cm that results in an image of 1 ’333 ’333 x 800’000 pixels that adds up to 1.66 TB of data. The credit for the 
production of the image goes to Mappuls AG / Axon Vibe AG. The complete dataset was made out of 7 ’800 aerial 
images at flight levels between 4 ’400 and 9 ’000 meters. The orthoimage can be experienced on foot using protective 
slippers or from an elevated positioned balustrade as shown in the following pictures. 



Fig. 5. The Swissarena exhibition room at the Swiss National Transport Museum in Lucerne Switzerland 


Since the opening of the permanent Swissarena exhibition different ideas to interact with the image where intro- 
duced. For instance, a trolley with a location laser pointer was used to allow visitors to get information about certain 
spots on the map (Fig. 6) or round plates with a section of the orthoimage on them are available so visitors could try 
to find the respective spots on the map and place the plates on the ground (Fig. 7). Using AR as a tool for more 
substantial interactions with the image seemed promising. 



Fig. 6. Trolley with display and laser pointer to explore the map Fig. 7. Overlay map puzzle using circular sections of the map 
(Courtesy of Swiss National Transport Museum Lucerne) (Courtesy of Swiss National Transport Museum Lucerne) 
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4 . Exploring AR markers using large-scale orthoimages 


Using internal sensors of mobile devices like compass, accelerometers and GPS can be used to create AR. However, 
the accuracy for the given use case cannot be met with currently integrated sensor technologies. AR using visual 
tracking, as in the presented case, requires natural or artificial images called markers that allow a camera device to 
position and orient itself relative to the coordinate system of the marker image as show in Fig. 8. This yields higher 
accuracy of AR tracking in a local space. 



Itlrltltlt 


Fig. 8. Tracking of artificial marker images (ArUco 2016) Fig. 9. Feature plot of natural marker images 

While the visual tracking of artificial markers, as shown above, can be very effective due to the well detectable and 
trackable nature of shapes like these, the tracking of natural feature markers poses problems when the image marker 
does not present a required number of trackable features. (Fig. 9 a). Important factors like contrast and noise of a 
marker image can strongly influence the possible number of calculable features. 

To effectively explore AR concepts and benchmark tracking performances of AR using aerial photography as 
markers, we introduced a test field of approximately 5 square meters on the floor of a public spot at the University of 
Applied Sciences and Arts Northwestern Switzerland (FHNW). While similar lighting conditions were present, the 
texture of the test print is different from the glossy surface of the original. The test print depicts a 50x50 km extract 
of central Switzerland. 



Fig. 10. Test print for research purposes at the FHNW in Muttenz 
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4.1 Generating a large AR trackable data set as a multi marker 


A requirement for the AR app was that there no additional visual markers should be used for localization and 
tracking except the orthoimage itself. When generating a robust set of markers distributed over the floor, the extremely 
large image dimensions as well as possible distractions and occlusions of the ground due to lighting and crowds of 
visitors had to be taken into account to cover the entirety of the orthoimage for tracking as shown in Fig. 5. 

The AR tracking and the resulting marker dataset was further required to not surpass manageable amounts of stor- 
age on average mobile devices as well as computational capacities of such devices. 

Another issue concerning the markers were the different distances from the device camera to the actual marker. 
This is due to the fact, that the orthoimage can be viewed from a ground level distance of around 1 m above, or from 
the balustrade with an approximate distance of 3-4 meters. This would influence the robustness of a tracked marker 
image significantly. 

A sparse quad tree based multi-marker approach was used to optimize on the matter of performance and different 
viewing distances. While the layout of the markers using a quad tree is systematic (Fig. 11), the distribution of the 
generated markers over the whole image data can be arbitrary. In our case it focusses on major cities and areas of 
interest all over Switzerland. Another factor for the tracking was the number of trackable features within a marker tile 
that would influence the robustness of the tracking as shown in Fig. 9. The better the quality of a single marker tile, 
the less LOD levels are required for a stable AR tracking. 

An original distribution of markers to be used as a combined multi marker was proposed by (Zwick 2016) and is 
shown in Fig. 12. While the graphic shows the sparse quad tree distribution, the rectangular markers themselves are 
of equal image dimension throughout all levels of detail. The goal was to minimize redundancy and subsequently the 
total number of tracked marker images for better performance. 



Fig. 11. Marker image quadtree layout (Zwick 2016) 



Fig. 12. Marker distribution (Zwick 2016) 


LODCH 

LODI 

LOD2 


5. The prototype AR app 


The resulting prototype was developed using Unity3D (Unity3D n.d.) as development framework and the Vuforia 
SDK (Vuforia n.d.) was chosen as AR tracking framework for visual marker tracking. This enabled us to deploy the 
app simultaneously on different platforms like Android and iOS at the same time see Fig. 13. This decision was based 
on the evaluation of different frameworks and technologies and provided a more feasible and more goal oriented 
solution than using other well-known technologies such as OpenCV or ARToolKit. 

The app is available for free to any visitor of the museum and is distributed over public platforms like App Store 
or Google Play Store. 
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Fig. 13. Prototype software architecture 


5.1 Geospatial modules and content 

Most visitors, according to the Swiss National Transport Museum, are families or school classes. The museum 
emphasizes on mobility and transport hence the chosen contents for the augmentation had to reflect on those topics. 
Given the premise to create an augmentation with geospatial information three categories of data where identified and 
integrated into the app as separate modules for user interaction. The chosen data sources where supposed to be open 
source wherever possible: 

- Module: Location 

Enable the user to spatial querying of location names for towns, lakes etc. which can be freely pointed at using 
a 3d cross-hair. 

Data source: 

Using OpenStreetMap geo location service Nominatim (OpenStreetMap n.d.) as source for reverse geocoding 
to allow the identification of locations on the map. 

- Module: Weather 

Showing the currently distributed weather situation over the country with symbols and simulated cloud cover- 
age to give the user an idea about live weather conditions. 

Data source: 

Weather data from (OpenWeatherMap n.d.) presents a large source of distributed weather data all over the 
world. 

- Module: Air traffic 

Using scaled 3D models of airplanes and simulated trails to represent an airplanes trajectory allows the user to 
monitor current movements with the national air space of Switzerland. Further information can be made avail- 
able when interacting with an object. 

Data source: 

Using a local network of transponder tracking data that covers most of the countries air traffic in real-time was 
provided courtesy of Swiss Flightlive (SwissFlightlive n.d.). 


To put the usage of spatial data into perspective Fig. 14 shows some of the integrated data layers. The AR tracking 
allows us to reverse geocode positions on the map since the device we are using also represents a spatial entity while 
we are operating it. 
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Fig. 14. Schema of spatial data layers in the swissARena AR App 


6. Results 

Augmenting spatial real-time data in this context enhances the experience of a user exploring the orthoimage on 
foot and our implemented examples show what the possibilities for further ideas could look like. The following figures 
give an impression of the results and have been captured directly from respective mobile devices. 



Fig. 15. Reverse geocoding module. Via touch input the center point of the screen determines the city, town or village on that exact point. 
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Fig. 17. Real-time air traffic shown as 3d models of airplanes with additional transponder information available 


As shown in Fig. 15 the user could point the device at a certain location marked by the visual indicator to determine 
the location targeted and thus give the module a quiz like feel. In Fig. 16 the live weather situation at distributed 
locations is shown and a particle cloud system simulates strongly covered areas. The third module is depicted in Fig. 
17 and shows live air traffic and some detailed information on a plane currently passing by. 
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7. Conclusions & Future Work 


With the described investigation on the use of large scale orthoimages as AR markers and the resulting prototype 
app we were able to show that using optimized marker datasets for the presented case provides a practical solution for 
mobile AR apps of this scale. The factors for enhancing the robustness of the tracking have been identified but also 
other random variables such as obstructions and lighting are playing a major part. 

While public feedback had been gathered occasionally over the past few months, there is no dependable data on 
user reception of the app and this data has yet to be gathered. One issue that was briefly emerging is that the navigation 
within the app should likely be optimized for a more intuitive interaction since using touch input in an AR context 
tends to be rather suboptimal. 

Expanding on this idea, we identify great potential for using this technology and setup for educational purposes. In 
the case of schools, floor maps would need to be replaced by printed maps of smaller dimensions, which have been 
successfully tested as part of this project. However, creating successful map-based AR apps in educational settings 
will require a close collaboration between geospatial, educational and design experts. 
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